Surgical Cockpit Comprising Multisensory and Multimodal Interfaces for Robotic Surgery and Methods Related Thereto

ABSTRACT

Local surgical cockpits comprising local surgical consoles that can communicate with any desired remote surgical module (surgical robot), for example via a shared Transmission Control Protocol/Internet Protocol (TCP/IP) or other unified open source communication protocol or other suitable communication system. The systems and methods, etc., herein can also comprise a modular approach wherein multiple surgical consoles can network supporting collaborative surgery regardless of the physical location of the surgeons relative to each other and/or relative to the surgical site. Thus, for example, an operator operating a local surgical cockpit can teleoperate using a remote surgical module on a patient in the same room as the surgeon, or surgeons located in multiple safe locations can telemanipulate remote multiple surgical robots on a patient in or close to a war zone.

PRIORITY CLAIM

The present application claims the benefit of copending U.S. ProvisionalApplication Ser. No. 61/315,018, filed Mar. 18, 2010, which applicationis incorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION

Despite advances in personnel, technology, and force protection, warfighters remain vulnerable to blast wounds, burns, and multiplepenetrating injuries not usually encountered in civilian settings. Thereis a fundamental need to deploy skilled personnel equipped with advancedtechnology to provide medical and surgical attention as close aspossible to the point of injury for a soldier in the battlefield andcivilians in a remote location or during natural or manmade disasters.Although war surgery is aimed to treat combat casualties at far forwardlocations and under austere conditions continues to save lives, thenature of battlefield injuries shortens the “golden hour” in whichhighly skilled medical attention is required to stabilize the soldier.

One of the major problems in the military and its integrated healthservices support system to triage, treat, evacuate, and return soldiersto duty is the occasional mismatch between the type of injury and thetype of surgeon available to treat it. For example, a heart surgeon maybe required to perform a craniotomy. Telemedicine in general andtelerobotic surgery in particular are a means to mediate the narrowspectrum of available front line surgical expertise.

There has gone unmet a need for improved methods that provide one ormore of the needs indicated above, for example robust teleroboticcapabilities so an expert surgeon could perform critical steps of anoperation from the continental U.S. or other desired location while thesoldier or other patient is located anywhere around the globe.

The present systems and methods, etc., provide these and/or otheradvantages. Present systems and methods, etc., extend the spectrum ofsurgical expertise to a level appropriate for the type and nature ofbattlefield wounds. The present systems and methods, etc., also extendto any desired area of surgery including those well beyond the militaryarena.

SUMMARY

The present systems and methods, etc., comprise local surgical cockpitscomprising local surgical consoles that can communicate with any desiredremote surgical module (surgical robot), for example via a sharedTransmission Control Protocol/Internet Protocol (TCP/IP) or otherunified open source communication protocol or other suitablecommunication system. The systems and methods, etc., herein can alsocomprise a modular approach wherein multiple surgical consoles cannetwork supporting collaborative surgery regardless of the physicallocation of the surgeons or other operators relative to each otherand/or relative to the surgical site. Thus, for example, a surgeonoperating a local surgical cockpit can teleoperate using a remotesurgical module on a patient in the same room as the surgeon, orsurgeons located in multiple safe locations can telemanipulate remotemultiple surgical robots on a patient in or close to a war zone or anyremote location.

One aspect of the systems, methods, etc., herein is to provide amultisensory, multi-modal surgical workstation (surgical cockpit). Thisworkstation can provide a command post allowing the surgeon(s) tovisually immerse themselves into the remote surgical space. It providesperipheral information such as vital signs, as well as visual feeds fromthe operating room (OR) surrounding the actual surgical site on or inthe patient. In this way, the surgical console provides high situationalawareness as well as the capability to dynamically interact with theother functions of the OR. This is accomplished by multiple streams ofvisual, audio, and kinesthetic inputs. Special attention can also paidto avoiding information overload of the surgeon.

In a further aspect, the present methods, devices, systems, etc., arerelated to a local surgical cockpit comprising a base, a frame disposedon the base, a seat for an operator disposed on the frame, and a remotesurgical console configured such that the operator can operate theconsole for remote surgery while in the seat, wherein the seat can beergonomic and can be operably connected to the frame such that the seatcan be retainably tilted from a substantially upright position to asubstantially supine position.

In certain embodiments, the seat can comprise an independently movableheadrest, backrest, seating plate and footrest, the seat furthercomprising positioning elements operably connected to the independentlymovable headrest, backrest, seating plate and footrest and providing atleast three axes of retainable positioning movement for each of theindependently movable headrest, backrest, seating plate and footrest.The seat can comprise a lumbar support comprising retainable positioningmovement for support of the lower back. The reference body posture ofthe seat can correspond to a human body posture that can be fullyrelaxed in micro gravity.

The cockpit further can comprise at least one peripheral device operablyconnected to move with the seat when the seat is moved so that thelocation of the peripheral device relative to the operator in the seatis substantially unchanged. The peripheral device can be at least one ofa monitor facing an operator in the seat and operably linked to displaya remote surgical site, a heads-up display disposed in front of thelocal surgeon's eyes, and an input device disposed at a hand of theoperator and operably linked to provide input to a corresponding devicelocated at the remote surgical site.

Another aspect comprises a local surgical cockpit comprising a localsurgical console configured for transmitting surgical movements of localsurgeon operating the local surgical console to a remote surgery site,and a head-mounted display disposed in front of the local surgeon's eyesin surgical position in the cockpit to operate the console for surgery,wherein the head-mounted display can be configured to depict at leastimages of a remote surgical site under remote operation by the operator.The local surgical cockpit can be part of a system and the systemfurther can comprise remote image sensors operably connected to thehead-mounted display to transmit the image of the remote surgical site.The head-mounted display can extend to the local surgeon's eyes from anarticulated boom or other retention structure disposed in front of thelocal surgeon's eyes, which retention structure can be actuated by atleast one hand control located on the cockpit, or by voice control orotherwise as desired. The head-mounted display can be disposed on ahead-mounted frame configured to rest on an operator's head and tomaintain the images in front of the local surgeon's eyes when theoperator's head moves. The head-mounted display can comprise twoseparate streams of video displayed to each eye of the local surgeon'seyes, each stream comprising corresponding right and left eye views of aremote surgical site to provide a 3-D image of the site.

The cockpit further can comprise at least one monitor operably held tothe base of the cockpit, and the head-mounted display can comprise twoseparate streams of video displayed to each eye of the local surgeon'seyes, each stream comprising corresponding right and left eye views of aremote surgical site to provide a 3-D image of the site. The monitor(s)can also be 3-D.

In another aspect, the systems, etc., are directed to a local surgicalcockpit comprising a local surgical console configured for transmittingsurgical movements of an operator operating the local surgical consoleto a remote surgery site, and comprising at least one image displaydevice configured to depict at least one image of the remote surgicalsite, the display device further depicting augmented reality for theoperator comprising augmented information shown on the display andsuperimposed over the image of the remote surgical site.

The local surgical cockpit can be part of a system and the systemfurther can comprise remote image sensors operably connected to thehead-mounted display to transmit the image of the remote surgical site.The augmented information can comprise at least one of preselectedmargins to dissect during the surgery and a mask of vital structures inthe remote surgical site overlaid over the images of the remote surgicalsite. The display device can further display further augmentedinformation either to a side of or superimposed over the image of theremote surgical site and the further augmented information can compriseat least one of blood pressure, temperature, O₂ level, CO₂ level,intracranial pressure, a preplanned trajectory for a surgical tool, tooltype, suction on/off, a bottom task bar, recording capabilities, currenttime, and elapsed time. The image of the remote surgical site and theaugmented information can comprise blending graphical images withreal-world views of the remote surgical sit, and can be provided by atleast one of an endoscopic camera, a remote surgical site camera, or acamera showing an operating room.

In still another aspect, the systems, etc., are directed to a localsurgical cockpit comprising a local surgical console configured fortransmitting surgical movements of an operator operating the localsurgical console to a remote surgery site, and comprising a localsurgical instrument comprising local input surgical fingers configuredto provide input to corresponding remote surgical fingers configured tomanipulate a remote surgical instrument at a remote operation site,wherein the local fingers can be high frequency haptic fingersconfigured to provide tactile feedback to the operator based onacceleration of the remote surgical instrument manipulated by the remotesurgical fingers.

The local surgical cockpit can be part of a system and the systemfurther can comprise the remote surgical fingers, and wherein the remotesurgical fingers can be haptic fingers configured to provide tactilefeedback to the operator based on acceleration of the remote surgicalinstrument manipulated by the remote surgical fingers.

The local surgical cockpits can be configured such that operators indifferent locales can operate simultaneously on a single surgical site;such that operators can relieve each other in a single surgery at asingle surgical site; or to provide a teaching surgical cockpit and astudent surgical cockpit providing haptic feedback to a student operatorgenerated by a teaching operator. The haptic feedback to the student cancomprise movements of a remote surgical instrument controlled by theteaching operator or tactile feedback from a surgical site beingoperated on by the teaching operator.

In a further aspect, the systems, etc., are directed to a local surgicalcockpit comprising a local surgical console configured for transmittingsurgical movements of an operator operating the local surgical consoleto a remote surgery site, and comprising at least seven degrees offreedom for a local surgical instrument manipulated by a robotic armmanipulated by the operator, wherein the console can be configured suchthat the seven degrees of freedom can be transmissible to a remotesurgical instrument located at a remote surgical site and manipulated bythe operator operating the console.

The local surgical cockpit can be part of a system and the systemfurther can comprise the remote surgical instrument operably connectedto the local surgical instrument such that the remote surgicalinstrument precisely responds in at least seven corresponding degrees offreedom to movements of the local surgical instrument. The degrees offreedom can comprise at least nine degrees of freedom for the localsurgical instrument manipulated by the operator and a corresponding ninedegrees of freedom for the remote surgical instrument. The degrees offreedom can comprise at least twelve degrees of freedom for the localsurgical instrument manipulated by the operator and a correspondingtwelve degrees of freedom for the remote surgical instrument, whereinthe local robotic arm can comprise a shoulder joint, an elbow joint, awrist joint and the three fingers, each comprising at least thefollowing degrees of freedom: shoulder can comprise 2 degrees offreedom; elbow can comprise 1 degree of freedom; wrist can comprise 3degrees of freedom; the three fingers can comprise 2 degrees of freedomeach.

The local surgical instrument can comprise at least three input fingersconfigured to provide input to a corresponding at least three remotesurgical fingers configured to manipulate a remote surgical instrumentat a remote operation site, wherein the at least three input fingers canbe configured to be manipulated by a single hand of an operatoroperating the local surgical instrument, and wherein the at least sevendegrees of freedom can comprise at least two degrees of freedom for twoof the three remote surgical fingers and at least three degrees offreedom for a third of the three remote surgical fingers, or the degreesof freedom can comprise at least nine degrees of freedom comprising atleast three degrees of freedom for each of the three remote surgicalfingers.

The local robotic arm can comprise a shoulder joint, an elbow joint, awrist joint and the three fingers, each comprising at least thefollowing degrees of freedom: shoulder can comprise 2 degrees offreedom; elbow can comprise 1 degree of freedom; wrist can comprise 3degrees of freedom; the three fingers can comprise 2 degrees of freedomeach. Or the three fingers can comprise 3 degrees of freedom each.

The degrees of freedom provide for variable desired positioning andorientation of a tip of the remote surgical instrument in space in 6parameters including Cartesian position (x,y,z), and angular orientation(x y z θ, θ, θ). Control of the remote surgical instrument further cancomprise scaling factors configured such that motion input by theoperator can be attenuated or amplified with respect to the remotesurgical instrument. Control further can comprise indexing configured toallow the operator to disengage the input device from the remotesurgical instrument to reposition his/her arms and engage again.

In still yet another aspect, the systems, etc., are directed to a localsurgical cockpit comprising a local surgical console configured fortransmitting surgical movements of an operator operating the localsurgical console to a remote surgery site, and comprising a localsurgical instrument comprising at least three input fingers configuredto provide input to a corresponding at least three remote surgicalfingers configured to manipulate a remote surgical instrument at aremote operation site. The at least three input fingers can beconfigured to be manipulated by a single hand of an operator operatingthe local surgical instrument.

The local surgical cockpit can be part of a system and the systemfurther can comprise the three remote surgical fingers operablyconnected to the three input fingers such that the three remote surgicalfingers precisely respond to movements of the three input fingers.

The at least three input fingers can be configured to correspondrespectively to a) an operator's thumb, b) an operator's index andmiddle fingers, and c) an operator's ring and little fingers; or to a)an operator's thumb, b) an operator's index finger, and c) an operator'smiddle, ring and little fingers. The at least three input fingers can behaptic fingers configured to provide tactile feedback to the operatorbased on acceleration of a remote surgical instrument manipulated by theremote surgical fingers. The three input fingers can be operablyconnected so that two fingers control remote surgical fingers and theremaining third finger controls an external device, which can be a oneor more of an electrocautery device, a laser photocoagulator, a stapleapplier or other device as desired. The external device can also be anoptical aspect of the camera system such as focus, zoom, rotation, orfield-of-view.

In another aspect, the systems, etc., are directed to a local surgicalcockpit comprising a local surgical console configured for transmittingsurgical movements of an operator operating the local surgical consoleto a remote surgery site, and comprising at least one haptic foot pedalconfigured to be operably connected to at least one remote device at aremote operation site, wherein the at least one haptic foot pedal can beconfigured to be manipulated by a foot of the operator operating thelocal surgical console to cause a movement or control change in theremote device.

The local surgical cockpit can be part of a system and the systemfurther can comprise the at least one remote device operably connectedto the at least one haptic foot pedal. The local surgical cockpitfurther can comprise at least two haptic foot pedals configured to beoperably connected to the at least one remote device at the remoteoperation site. The local surgical cockpit further can comprise at leasteight haptic foot pedals configured to be operably connected to at leasttwo remote devices at the remote operation site, the at least eighthaptic foot pedals divided to provide at least a first foot pedal setand second foot pedal set, wherein a first foot pedal set and secondfoot pedal set can be each disposed to be manipulated by a right foot ofthe operator and by a left foot of the operator, respectively.

If desired, for each of the first foot pedal set and second foot pedalset, the sets can each contain four pedals with each of the four pedalsin one of four quadrants of a circle. The pedals can also be set in anarray. Opposed or otherwise set off pairs of pedals can be assignedopposed functions at the remote surgical site. The opposed functions canbe suction and irrigation. The four pedals can also be assignedcomplementary functions for a remote instrument at the remote surgicalsite. The four pedals can control the viewing angles of an endoscopiccamera. The local surgical cockpit further can comprise a dead zone thatprevents two opposing functions being implemented simultaneously.

The at least one haptic foot pedal can also control at least one ofcamera angle, camera zoom, camera focus, irrigation, suction, robotbrakes, electric coagulation, laser photocoagulation.

In a further aspect, the systems, etc., are directed to a local surgicalcockpit comprising a local surgical console configured for transmittingsurgical movements of an operator operating the local surgical consoleto a remote surgery site, and comprising at least one virtual consolecontrol knob presented virtually to the operator and configured to bemanipulated by the operator to generate control signals for acorresponding remote device at a remote operation site.

The local surgical cockpit can be part of a system and the systemfurther can comprise the corresponding remote device. The at least onevirtual console control knob can be a binary switch configured toprovide on/off signals to the corresponding remote device. The at leastone virtual console control knob can be a gradual control knobconfigured to provide gradual control signals to the correspondingremote device. The virtual control knob can be operably connected to oneof the three fingers of the haptic device. In some embodiments, thevirtual control knob must be virtually gripped by two or more fingers ofthe haptic device before it may be rotated.

In still yet a further aspect, the systems, etc., are directed to atleast two local surgical cockpits each comprising a surgical consoleconfigured for transmitting surgical movements of an operator operatingthe local surgical console to a distant remote surgery site locatedoutside at least one building containing at least one of the surgicalcockpits, wherein each cockpit can comprise a respective first andsecond set of at least two local robotic input arms configured toprovide input to corresponding first and second remote sets of at leasttwo corresponding remote robotic arms each configured to manipulate aremote surgical instrument at a single remote operation site, whereinthe respective first and second set of local robotic input arms can beconfigured to be manipulated by respective first and second surgeonsworking in concert on the remote surgical site.

The local surgical cockpit system can be part of a further system andthe further system further can comprise the first and second remote setsof at least two corresponding remote robotic arms. The distant remotesurgery site can be located outside any building containing any of thelocal surgical cockpits. The system can be configured such thatoperators in different locales can operate simultaneously on a singlesurgical site; such that operators can relieve each other in a singlesurgery at a single surgical site; and/or to provide a teaching surgicalcockpit and a student surgical cockpit providing haptic feedback to astudent operator generated by a teaching operator. The haptic feedbackto the student can comprise movements of a remote surgical instrumentcontrolled by the teaching operator or tactile feedback from a surgicalsite being operated on by the teaching operator.

In still yet a further aspect, the systems, etc., are directed to alocal surgical cockpit comprising a local surgical console configuredfor transmitting surgical movements of an operator operating the localsurgical console to a remote surgery site, and comprising at least fourlocal robotic input arms configured to provide input to a correspondingat least four remote robotic arms each configured to manipulate a remotesurgical instrument at a remote operation site, wherein the at leastfour local robotic input arms can be configured to be manipulated by atleast one surgeon operating the local surgical console.

The local surgical cockpit can be part of a system and the systemfurther can comprise the four remote robotic arms operably connected tothe four local robotic input arms such that the four remote robotic armsprecisely respond to movements of the four local robotic input arms. Thefour remote robotic arms can be held in a sole arm-retention structure,which can be configured to hold the four remote robotic arms such thatthe arms cannot collide with each other. The cockpit can be part of asystem comprising at least two local surgical cockpits each configuredfor an operator, and wherein the system can be configured such that eachoperator can simultaneously hold a single remote robotic arm, or suchthat the operators can switch control of a remote robotic arm betweeneach other.

In another aspect, the systems, etc., are directed to a local surgicalcockpit comprising a local surgical console configured for transmittingsurgical movements of an operator operating the local surgical consoleto a remote surgery site, and comprising local 3-dimensional audioconfigured to obtain remote 3-dimensional audio input from a remoteoperation site and provide corresponding local 3-dimensional audio to anoperator operating the console.

The local surgical cockpit can be part of a system and the systemfurther can comprise remote 3-dimensional audio sensors operablyconnected to the local 3-dimensional audio such that the local3-dimensional audio precisely transmit 3-dimensional audio signals fromthe remote 3-dimensional audio sensors. The 3-dimensional audio signalscan be correlated with tactile feedback to provide correlated responseto haptic input devices at the local surgical cockpit.

These and other aspects, features and embodiments are set forth withinthis application, including the following Detailed Description andattached drawings. Unless expressly stated otherwise, all embodiments,aspects, features, etc., can be mixed and matched, combined and permutedin any desired manner.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an isometric view of an exemplary cockpit and consoleaccording to various aspects and features discussed herein.

FIG. 2 is a view of a fully relaxed body posture as observed inastronauts who are subjected to microgravity.

FIG. 3 is a second view of a fully relaxed body posture as observed inastronauts who are subjected to microgravity.

FIG. 4 is a third view of a fully relaxed body posture as observed inastronauts who are subjected to microgravity.

FIG. 5 depicts a side view of exemplary components of a seat asdiscussed herein.

FIG. 6 depicts a head-borne HMD comprising a headset.

FIG. 7 depicts a side view of a further exemplary cockpit and consoleaccording to various aspects and features discussed herein.

FIG. 8 depicts an isometric view of another exemplary cockpit andconsole according to various aspects and features discussed herein.

FIG. 9 depicts an isometric view of still a further exemplary cockpitand console according to various aspects and features discussed hereincomprising an array of eight intuitive haptic foot pedals (four for eachfoot) that allows the surgeon to control multiple devices and to switchthe control between them.

FIG. 10 depicts an isometric view of a three fingers haptic hand whereinthe middle and index fingers are lumped into a first port, the ring andthe fifth finger are lumped into a second port and the thumb is in thirdport.

FIG. 11 shows some of the degrees of freedom obtainable with athree-fingered hand.

FIG. 12 depicts a top view of a three fingers haptic hand wherein theindex finger is lumped into a first port, and the middle, ring and fifthfinger are lumped into a second port, and the thumb is in third port.

FIG. 13 depicts an isometric view of an articulated haptic arm coupledto a three fingers haptic hand.

FIG. 14 depicts an exemplary conceptual layout of the visual informationdisplayed by the cockpit to the surgeon

FIG. 15 depicts an isometric view of four robotics arms that can beteleoperated by one or two surgeons in multiple modes of operation

FIG. 16 is a block diagram depicting certain aspects of an exemplarysoftware architecture that can be used with the local surgical cockpitsand remote surgical procedures herein.

DETAILED DESCRIPTION

FIG. 1 depicts an exemplary cockpit and console according to variousaspects and features discussed herein. Cockpit 2 comprises a surgicalconsole 25 comprising screens, input devices and the like, and astructural frame 4 disposed on a base 6. The frame 4 provides physicalsupport, directly or indirectly through other components, to the consolecomponents of the cockpit 2 such as robotic arms 20 and head mounteddisplay 22 (HMD), and provides adjustable mounting capabilities forevery desired element. In other embodiments, various console elementsdiscussed herein can be disposed on other support structures instead ofthe frame 4, such as nearby walls, desks, tripod stands, etc.

The cockpits 2 herein can also comprise two or more different surgicalconsoles in one cockpit or two or more different surgical consoles intwo or more different surgical cockpits that are operably connected toeach other either locally or at the remote surgical site (or otherwiseas desired). This allows, e.g., surgeons or other operators incompletely different locales to operate or otherwise interactsimultaneously on a single surgical site.

The seat 8 is an adjustable ergonomic seat that positions the body ofthe surgeon in a desired position such as an optimal, selectable postureto reduce fatigue or other discomfort. The seat 8 allows for positioningand orientation of its components (e.g., headrest 16, backrest 14, seatplate 10, footrest 12, armrests 18) in any possible configuration fromsitting fully upright to a completely supine position and to accommodatedifferent body types. Thus, positioning elements of the cockpit 2 areoperably connected to the independently movable headrest 16, backrest14, seating plate 10, footrest 12 and armrest 18 to provide at leastthree axes of retainable positioning movement for each of thoseindependently movable elements relative to each other. For example, allsuch seating elements can be movable relative to the frame, or one canbe securely retained on the frame 2 and the other movable seatingelements can be movable. The reference body posture can be the oneadopted by the human body in microgravity.

As shown in FIG. 5 as well as FIG. 1 and other figures depicting thefull cockpit, the joint 26 between the stationary base 6 and the frame 4can be actuated to allow titling motion of the frame 4. The tiltingmotion accommodates the body position of the surgeon through differentangles with respect to gravity. The tilting angle can range from asupine posture where the gravity vector is perpendicular to the spine,to a full upright posture where the gravity vector is parallel to thespine. During the tilting of the entire frame 4 as a whole, the relativeposition and orientation of the surgeon's body segments as well as therelative position and orientation with respect to the peripheralelements can be maintained. The backrest 14 of seat 8 can also comprisea variable lumbar support 96 that provides retainable positioningmovement for support of the lower back.

As shown in this embodiment, the frame 4 is articulatable and providesthe ability to removably attach and adjust peripheral devices such asmonitors 24, HMDs 22 and forearmrests 18 and input devices 34 and forsuch peripheral devices to articulate in unison with respect to astationary base 6 and/or seat 8. For example, when the seat position ismodified, the location of the display and input device 34 moveaccordingly so that the relative position of the peripheral devices tothe surgeon stays substantially the same. The adjustable elements of theseat allow body posture adjustments automatically or on demand. In theautomatic mode, the chair positioning can be under the control of ahigh-level software module. The seat and other positioning elements canbe moved electronically or mechanically via motors, manually, orotherwise as desired.

Articulated mechanical linkages and interfaces can be provided in theframe 4 for desired subsystems such as (1) two sets of articulatedlinkages 28, 30 for attaching the displays 50 such as an array ofscreens or monitors (one such monitor 24 is shown in FIG. 1 and FIG. 2connected to articulated linkage 28) and the HMD 22, which is connectedto HMD articulated linkage 30; (2) two interfaces 32 to support the armsand the input device 34; (3) a footrest support interface 36 to supportthe foot pedals 38 and the footrest 12; and, (4) a seat interface 40 tosupport the seat 8. All of these interfaces can be fully adjustable.Additional interfaces include headrest support interface 42 and backrestsupport interface 44. Friction based mechanical joints, or othersuitable connectors, lock the cockpit 2 in the desired position.Interfaces that change their position frequently, such as interface 28for the HMD 22, can be locked in place by electro-mechanical brakes.

A surgeon can be required to perform high dexterity manipulation duringthe course of surgery that may last for several hours. The chair can bedesigned such that it can be adjusted to emulate the same body postureas observed in astronauts who are subjected to microgravity (see FIGS.2-4). In this body position the muscles reach their rest length and thusreduce potential fatigue. Moreover, the capability to adjust the bodyposture can be needed to avoid postural fixity, promote bloodcirculation, reduce joint pressure and muscle tension, and increasesituational awareness

In addition to the body posture, arm position affects the ability of thesurgeon to control the input device 34 to the surgical robot. Armmanipulability can be a term that can define mathematically how jointangles (shoulder and elbow) are mapped into the hand position. It can beshown mathematically and proved experimentally that in order to maximizethe manipulability, the elbow joint angle should be about 90°.Interestingly, this elbow joint angle can be also adopted by the humanbody in microgravity (see FIGS. 2-4—the elbow angle can be 92°+/−15°).Following this rationale, the arm of the surgeon can be positioned usingadjustable armrests 18 with the same angles as indicated in FIGS. 2-4 tomaximize the manipulability of the arm.

Footrest 12 supports the feet in a similar fashion to FIGS. 2-4 giventhe same rationale. This can enhance the ergonomic interactions betweenthe surgeon's feet and hands with the controls and peripheral equipment.Further, the surgeon's foot can be fixed in space and ankle movementscan be used to activate the pedals surrounding it. Functions such ascontrolling the cameras can be implemented by linking the ankleflexion/extension to the camera pitch movements and ankle rotation tothe camera yaw movements. Haptics can be added to the pedals. Forexample, force feedback applied through the pedals can be correlatedwith camera position, or with irrigation or suction pressure.

Thus, the seat 8 can be adjusted to adapt to different bodies andchanging body posture, automatically or on demand. This can avoidpostural fixity, promote blood circulation, reduce joint pressure andmuscle tension and increase situational awareness.

FIG. 1 shows a frame-mounted HMD 22; FIG. 6 depicts a head-borne HMD 22comprising a headset 90. The embodiment in FIG. 6 comprises a head strap46 and a vertical strap 48 and 3-D audio input device 80. The display 50is disposed in front of the eyes of a user.

The display (HMD or otherwise such as a frame-borne monitor) can bedivided into two components: (1) the hardware and (2) the informationlayout. From the hardware perspective, one or more, even several, 2-Dand/or 3-D modalities can be used, for example two screens withprojected mirrors, a Head Mounted Display with two separate streams ofvideo displayed to each eye or a single 3-D screen, e.g., 120 Hz, withactive and synchronized shutter glasses at 60 Hz (Nvidia 3-D Vision).For analyzing the graphical information available to the surgeon in theOR, an exemplary layout for a display 50 is summarized in Table 1 anddepicted in FIG. 14:

[Top Left] [Main Display] [Top Right] 3-D remonstration of the target3-D display of Surgical Site 2-D display of Mentoring InformationAugmented Information: Augmented Information (on/off): (e.g., textbookanatomy, model or video  Preselected Margins to dissect  Blood Pressureclip of an expert performing the procedure,  Masks of vital structures Temp remote collaborator)  O2 SAT [Bottom Left]  CO2 [Bottom Right] Intracranial pressure 2-D display of MRI CT scans (can Preplannedtrajectory 2-D display of OR overview be browsed)  Tool Type  SuctionOn/Off Bottom Task bar  Recording capabilities  Time  Elapsed time

As can be seen in the Table and FIG. 14, the elements shown can includethe remote surgical site 52, remote operating room 54, remote target 56and remote surgical device 82 as well as other desired information. Suchinformation can be presented in segments of a single screen or onmultiple screens.

More specifically, FIG. 14 depicts an exemplary conceptual layout of thevisual information displayed by the cockpit 2 to the surgeon. Thislayout translates the verbal specification and description listed inTable 1. High level software of cockpit 2 can manage the content of thedisplay 50. The central view of the remote surgical site 52 point can bekept clear of overlays such as augmented reality by default if desired,with display of specific components under control of the surgeon.Information will flow to this display 50 from the endoscope/camera(s)targeting the remote surgical site 52 and remote target 56 within suchsite. In some embodiments, the display in FIG. 14 as well as hapticfeedback and other information from the remote surgical site and/oroperating room are transmitted to multiple local surgical cockpits. Thisallows, e.g., surgeons in completely different locales to operatesimultaneously on a single surgical site with the same visual display,as depicted in FIG. 14, or with visual displays containing identicalcore information as well as additional custom information as desired bythe surgeon. This also allows, e.g., surgeons in the same or differentlocales to relieve each other during a single surgery and for trainingsurgeons with common tactile and visual feedback through the visualdisplays and haptic input devices, etc.

In this example the dissection plane can be shown as a deformable blueline; this direction line can be defined during the preoperative stageand can be tracked during surgery by the high level software. Forcereflection signatures can be acquired as the position error between theinput device 34 and the remote surgical instrument 58 by the high levelsoftware module from a low level software module, discussed furtherbelow. This information can be presented as colored dots attached to thesurgical instruments (as overlays) to indicate safe (green) and unsafe(red) contacts for tissue resection. Patient vital signs can beoptionally shown in the central view or docked in the display margins.This peripheral information can be acquired by the high level softwaremodule and present visually to the surgeon. Recording capabilities canbe embedded into the main view and controlled by the high level softwaremodule allowing the surgeon to record the entire operation or individualsegments (with chapter markers set by the surgeon if desired) to form adetailed medical record of the procedure as well as broadcast forteaching and remote conferencing. The additional monitors 24 presentinformation acquired by imaging modalities along with an overview of theoperating room or other information as desired. The nature of theinformation can be procedure specific.

Returning to FIG. 1 and also referring to FIGS. 7-9, which depictvarious alternative cockpit embodiments and arrangements, the display 50can be a modular display integrated into the surgical cockpit 2 with thefollowing capabilities:

-   -   2-D and 3-D high definition visual displays (e.g., 1080        progressive lines) arranged in an array of 2-5 monitors 24. FIG.        9 shows an example with multiple monitors 24.    -   Head mounted display 22. The HMD can include two miniature        screens located 2 cm away from each eye and fed by two separate        streams of video signals to generate a stereoscopic image of the        surgical scene. The HMD 22 can be mounted on an articulated boom        112 as in FIGS. 1 and 7-9, and can provide a redundant display        to the flat panel array. In some embodiments, the HMD 22        displays only the remote surgical site 52, for situations where        the surgeons wish to focus on the surgical site exclusively. The        HMD can be mounted on any other suitable system such as a        headset 90 as in FIG. 6 or other mechanisms such as several        cables or parallel arms.    -   Synthesized display of multiple video sources fed from the        endoscopic camera or surgical site camera, a camera of the OR,        and along with applications presenting imaging information.

Desirable elements in the display design can include (1) reducedcognitive load for the surgeon, (2) support for surgical “flow”, and (3)ultimately, increased patient safety. The displays, and other elementsherein, can be used for both normal surgeries and for microsurgeries.Moreover, the microscope traditionally used for microsurgery may bereplaced, such as with a 1-2 mm scope. This change provides a widerfield of view.

In certain embodiments, the multiple (e.g., five) monitor layout and theHMD 22 present information in 3-D and provide support for variousvisualization, communication, and surgical performance functions,including (1) surgical instrument and the surgical scene—sensoryinformation can be displayed for smart tools with embedded sensors; (2)two-handed haptic clinical information e.g. compression tension; (3)“augmented reality” blending graphical images with real-world views andreal robot slaves e.g. go/no-go zones; (4) case archiving andvideoconferencing for guiding and collaborative purposes. In order toprovide a clean and informative interface while managing the cognitiveload of the surgeon, every pieces of information presented on thedisplay 50 can be called up or suppressed by the surgeon. This approachallows a custom display of information that can be dynamically changedduring the surgical procedure.

Additional display 50 modalities include large field-of-view domeprojection displays, which afford significantly larger display “realestate,” and high-resolution autostereoscopic displays.

The augmented reality can also provide numerical models to indicatecurrent tissue stresses propagating from instrument manipulations. Theinterface display, robot control commands, and/or audio can be recorded(with chapter markers set by the surgeon if desired) to form a detailedmedical record of the procedure as well as broadcast for teaching andremote conferencing.

For example as shown in FIG. 9, the cockpit 2 can include at least twofunctional interfaces for the surgeon's hands (master robotic arm) andthe feet (foot pedal array). As shown in FIG. 9 and other figures, theinterfaces can include two robotic arms 20 and eight foot pedals 28 intwo groups or sets of four. The surgeon controls all the specificfunctions of the surgical robot through these controls. The localcontrols also transmit force feedback to the surgeon as the remotesurgical tools interact with the target tissue. Separate interfacescontrol peripheral devices such as positioning/zooming the camera,camera angle, camera focus, suctions/irrigation, robot brakes, andcautery including electric coagulation, laser photocoagulation,stapling, etc.

In some embodiments, as shown in FIG. 9, the first and second foot pedalsets can each contain four foot pedals 38. In one arrangement, as shownin front of the left foot of the seated figure in FIG. 9, the fourpedals in can be arrayed in an arc in front of the user's foot. Inanother arrangement, as shown in front of the right foot of the seatedfigure in FIG. 9, the four pedals can each be in one of four quadrantsof a circle, typically 90° apart. Opposed pairs can be assigned opposedfunctions at the remote surgical site, such as suction and irrigation.Each of the four pedals can also or alternatively be assignedcomplementary functions for a remote instrument(s) at the remotesurgical site such as the viewing angles, focus, zoom, etc., of anendoscopic or operating room camera robot brakes, cautery such aselectric coagulation or laser photocoagulation, etc.

The following discussion is directed to an exemplary individual inputdevice 34, namely a robotic arm 20.

Turning to FIGS. 10-13, the master input device 34 is a multi degree offreedom (DOF) haptic device including two subsystems: (a) an articulatedhaptic arm 62 and (b) a three fingers haptic hand 64. Three capabilitiesthat input device 34 typically includes to facilitate the fundamentalcontrol of surgical tools by the surgeon through the cockpit 2 comprise:(1) positioning and orientation of the tool tip in space requires 6parameters—Cartesian position (x,y,z), and angular orientation (x y z θ,θ, θ); (2) scaling factors introduced such that the motion of thesurgeon hands controlling the input device 34 can be scaled down(attenuate) or up (amplify) with respect to the robot; and, (3) indexing(“clutching”), which allows the surgeon to disengage the input device 34from the robot to reposition his/her arms and engage again.

Two input devices 34 are typically fabricated and integrated into eachcockpit 2—one for each hand of the surgeon. The arm 62 of the inputdevice 34 can include six or seven DOF or more. The arm can beconstructed as a cable actuated SCARA-based machine (SCARA—SelectiveCompliant Articulated Robot Arm), or otherwise as desired (as with allcomponents of the systems discussed herein, unless specifically statedotherwise, the specific materials, manufacturing methods, etc., for thecomponents can be selected to optimize particular features andcharacteristics. Thus, the components can be made of steel, carbonfiber, etc., so long as the composition is acceptable for the desiredpurpose).

A cable actuated system can be common practice in designing hapticdevices. It allows for location of the actuators on a stationary base 6to transfer torques to each one of the joints through a system ofpulleys and cables. This configuration leads to a lightweight, lowinertia, low friction, and back drivable haptic device that can reflectback to the user the forces generated when the surgical robot interactswith tissues. Back drivability can be an important characteristic of anyforce feedback haptic device. It generates a negligible effect ofresistance as the operator moves the input device 34 in free space whichcan be the desired response as the surgical robot does not interact withany tissue.

The actuation and position sensors can be supported by a system ofamplifiers, along with low level software modules incorporating servofeedback loops for tele-operation and force feedback control algorithms.

If actuators with high gear ratios are introduced to the system, theuser feels the reflected inertia along with the friction in thegearbox—forces that mask the smaller effect of force feedback generatedas a result of the interaction of the surgical robotic tool interactingwith the tissue. The SCARA-based mechanism can be a classical roboticarm configuration which includes three consecutive axes with a rotationaxis perpendicular to the ground or parallel to the gravitationalvector. As a result, gravitational loads are not fully reflected intothe actuated joint and most of the load can be supported by thestructure elements of the haptic device and not by the actuatorassociated with the joint—a situation that leads to use of smalleractuators for the joint.

Force feedback can be eliminated from the operational mode of the systemif so desired. In such a situation, actuators are still typicallyincorporated into the tactile input device 34. Their secondary can be topreserve the registration between the input device 34 and the remotesurgical tool by locking the orientation of the tip of the input device34 (arm) once indexing is taking place. Indexing should typically onlyallowed for repositioning the end effector of the input device 34 in theCartesian space (x,y,z) through translation or repositioning. Theorientation of the input device 34 during the indexing process musttypically be preserved. Reorientation is not typically allowed duringthe indexing process in order to preserve the registration between theinput device 34 of the cockpit 2 and the surgical tool attached to thesurgical robot.

Force feedback capabilities can be achieved by a cable driven mastermechanism with a set of actuators (e.g., brushed DC motors) and positionsensors (encoders and potentiometers) attached to its base. Brushless DCmotors have a minor advantage compared with brushed motors as far astorque to weight ratio. However the high numbers of electrical wires formotor commutation may cancel out their minor advantage.

For example, the actuators can be selected to generate the followingpeak forces and torques: (1) translational forces 67 N, grasping force42 N, torques 2.4 Nm. The actuation and position sensors can besupported by a system of amplifiers, along with newly developedlow-level software modules incorporating servo feedback loops forteleoperation and force feedback control algorithms.

In another example, the input device 34 comprises: (1) the directkinematics of the haptic arm defined by mapping the joint angle to theend effector (hand interface), (2) The Jacobian matrix can be derived bymapping the joint angular velocity to the end effector velocities, (3)the manipulability as a performance measure can be defined, (4) a costfunction can be defined taking into account the manipulability measureand the link length of the mechanism. Using a brute force numericalsolution, for example, the cost function can be calculated across anentire workspace of different combinations of link lengths, for exampleones that have maximal dynamic manipulability with the minimal linklengths within a workspace of 10×10×10 cm.

The surgeon's hands interact with the master devices through athree-finger mechanical interface such as the three fingers haptic hand64. In FIG. 10, the middle and index fingers are lumped into the firstport 72, the ring and the fifth finger are lumped into the second port74 and the thumb is in third port 76. In FIG. 12, the index finger isput into the first port 72, the middle, ring and the fifth finger arelumped into the second port 74 and the thumb is in third port 76.

The 3 fingers interface mediates the significant gap between the 5fingers (5 fingers×4 DOF per finger=20 DOF) of the human hand and 2fingers systems having only 1 DOF. FIG. 11 shows some of the degrees offreedom obtainable with a three-fingered hand. Thus, the three fingersinterface allows a wide spectrum of control capabilities by thesurgeon's hand over a remote surgical instrument or other remote device.

The haptic hands herein can switch between various user-fingerconfigurations to maximize dexterity as desired. Each of the tactiledevices such as the robotic arm 62 and three fingers haptic hand 64 cancomprise 12 DOF or more.

Information perceived through the human sense of touch (haptics) can beclassified into two categories, cutaneous and kinesthetic. Cutaneousinformation can be provided via the mechano-receptive nerve endings inthe glabrous skin of the human hand. It can be primarily a means ofrelaying information regarding smallscale details in the form of skinstretch, compression and vibration. Kinesthetic sensing encompasseslarger scale details, such as basic object shape and mechanicalproperties, for example, compliance. This can be achieved via feedbackfrom the muscular and skeletal system.

If desired, not all DOF need have tactile (haptic) feedback. Forexample, only 3 out of the 6 DOF that can be sensed for the threefingers mechanism may include force feedback. Thus, in some embodiments,there will be a reduction from 12 potential actuated DOF in the arm-handcombination to only 9 fully actuated DOF with force feedback at the hand(fingers). For example, the remaining 3 non-actuated DOF will can beposition information that can be provided by the low frequency motion ofthe hand and the arm, cutaneous information can be provided to thesurgeon regarding tissue texture via high frequency actuatorsincorporated into the finger pad interface of the 3 fingers hand. Lowfrequency indicates forces provided in the range of 0-100 Hz; highfrequency indicates forces provided in the range of 100 Hz and higher.

The following functions were identified for the third finger: microscopecontrol, suction, irrigation, drilling, clutching. In order to providefull control of these functions, two types of buttons can beimplemented: (1) binary button (On/Off); (b) gradual knob(Volume/Magnitude).

Master Device DOF with Force Joint DOF Feedback Shoulder 2 2 Elbow 1 1Wrist 3 3 3 Fingers 6 3 Total 12 9

In certain embodiments, each finger has force feedback on 2 out of the 3DOF allowing flexion/extension movements and feedback. The fingers'adduction/abduction movements can be supported by a passive DOF with noforce feedback.

The three-finger design is particularly useful for use with (1) virtualknobs and switches 78 (such as shown in FIG. 14). For example, tocontrol retractors, electrocoagulators, or view of the scene and (2)anticipating new tool designs in the future, it allows the surgeon toregain the level of dexterity and manipulability of the human hand usedin open surgery but lost in current robotic systems. The virtual controlknob can be operably connected to one or more of the three fingers ofthe haptic input device(s). The virtual control knob can also beconfigured so that it must be virtually gripped by two or more fingersof the haptic device before it may be rotated or otherwise manipulated.

FIG. 9 depicts a cockpit 2 comprising an array of 8 intuitive hapticfoot pedals (4 for each foot) that allows the surgeon to controlmultiple devices and to switch the control between them. Two desirablepedal configurations: (1) serial arrangement (similar to a car) and (2)spatial arrangement in which 4 pedals surround each foot such that byflexing/extending the ankle and rotating the feet left/right all 4pedals are accessible. The serial arrangement can be limited to 3 pedalsfor each foot. Increasing the number of pedals beyond three makes itdifficult for the surgeon to locate the pedals while being immersedvisually in the surgical site. The spatial arrangement of 4 pedalssounding the feet from 4 different orthogonal directions provides aneasier registration between the feet and the pedals and a richer mediumas an input device 34.

The feet tactile interface can include a passive gimbal mechanism 84 tosupport the heel of the foot. The gimbal mechanism 84 allows the surgeonto move freely in any direction while avoiding gamble lock (the rotationin any direction will never exceed 90°). The four pedals can be arrangedaround the distal end of the feet in four orthogonal planes. Flexing theankle joint will press the top pedal while extending the joint willpress the bottom pedal. Moving the distal part of the foot left andright will press the left and right pedals.

A dead zone can be implemented in the design preventing a situation inwhich the feet activate two opposing functions simultaneously (e.g.,suction and irrigation). Each pedal can be controlled by a single DCservo motor. Through the software, the force displacementcharacteristics can be defined. Displacement characteristics of eachpedal provide the opportunity to change the function of this interfaceand assign functions to each pedal based on a specific operation(similar to the third finger of the hand interface). Moreover, thesoftware can allow the surgeon to change the nature of the pedal from anon/off switch to a gradual control switch.

The distribution of functions between the hand's third finger and thefoot can be assigned as desired. Functions such as controlling thecameras can be implemented by linking the ankle flexion/extension to thecamera pitch movements and ankle rotation to the camera yaw movements.Haptics can be added to the pedals. For example, force feedback appliedthrough the pedals can be correlated with irrigation or suctionpressure.

As noted previously, the cockpits 2 herein can also comprise two or moredifferent surgical consoles in one cockpit or two or more differentsurgical consoles in two or more different surgical cockpits that areoperably connected to each other either locally or at the remotesurgical site (or otherwise as desired). This allows, e.g., surgeons incompletely different locales to operate simultaneously on a singlesurgical site. This also allows, e.g., surgeons in the same or differentlocales to relieve each other in a single surgery at a single surgicalsite. This still further allows, e.g., exceptional training of surgeonsone by another, including providing tactile feedback to a studentsurgeon, for example from the movements of the surgical instrumentscontrolled by the teaching surgeon or to the student surgeon from asurgical site being operated on by the teaching surgeon.

FIG. 15 depicts an isometric view of a system comprising four remoterobotics arms 86 a-86 d holding remote surgical instruments 88 a-88 d ina remote operating room, which arms 86 can be teleoperated by one or two(or more) surgeons in multiple modes of operation, such as (1) solo by asingle surgeon from a local or a remote location (2) by two surgeons inwhich one or both are located locally with the robot or in one or tworemote sites.

The four robotic arms can contain a variety of peripheral devices and/orfunctions, such as an endoscope 100 comprising an endoscopic camera 102in remote surgical instrument 88 a coupled with positioning and/orzooming the camera, camera angle, camera focus. Similarly, remotesurgical instrument 88 b can contain a suction device 104 and irrigationdevice 106, and remote surgical instrument 88 d comprises a cauterydevice 108 for, for example, electric coagulation or laserphotocoagulation. The robotic arms can also be under the control ofrobot brakes 110 and can provide a staple applier 114.

The four arms 86 system duplicates two surgeons collaborating andsimultaneously interacting with the remote surgical site. The new systemprovides a new opportunity to explore collaborative surgery which tosome extent was not possible before due to the limited number ofavailable remote surgical arms 86. Moreover, the control over one or twoof the arms can be assigned to an artificial agent (software) and tofacilitate new methods of automation. In certain embodiments, anyrobotic arm can be assigned at any point by the primary surgeon to theother surgeon(s) or the artificial agent regardless of its location(local or remote).

In certain embodiments, the four remote robotic arms are held in a solearm-retention structure 116. The sole arm-retention structure can beconfigured to hold the four remote robotic arms such that the armscannot collide with each other.

The cockpit can be part of a system comprising at least two localsurgical cockpits each configured for an operator, and the system can beconfigured such that each operator can simultaneously hold a singleremote robotic arm, and/or so that the operators can switch control of aremote robotic arm between each other.

FIG. 15 also depicts an external, remote operating room camera 98. Thefunctions of this camera such as position, zooming, angle, focus, etc.,and other remote operating room devices and peripherals can also becontrolled by the surgeon or other operator in the surgical cockpit 2.

FIG. 6 depicts an exemplary headset 90 comprising a 3-D auditory inputdevice 80. The Auditory Interface can be any suitable 3-D auditorysystem such as a system comprising two sonic beam focused speakers. 3-Dauditory technologies include Hypersonic Sound (HSS) by AmericanTechnology Corp and Audio Spotlight by Holosonics. These technologiesallow a focal beam of sound to be transmitted into a specific point inspace (surgeon's ears) and avoid sound “pollution” in the operating room(OR). Both can still provide desired sound input from the operatingroom.

3-D auditory input devices 80 permit local surgical areas/cockpits tobecome consistent virtual listening areas without the pervasiveomnidirectionality of conventional loudspeakers. The high-precisiontargeting of directional beam of sound significantly minimizes thelevels of noise pollution in the local operating rooms while stillallowing the surgeon to respond to other sources of auditory inputs fromthe remote operating room.

The 3-D auditory input devices 80 can be used, for example, to conveythe following information via either natural or synthetic sound cues:(1) collisions between the surgical tools, (2) contact between the tooltip and some types of tissue, (3) stress levels applied to the tissue,(4) vital signs and emergency limits, (5) local pulse and vascular bloodflow to denote vessels that may need to be preserved or ligated.

There are several types of 3D audio effects, such as (1) widening thestereo image by modifying phase information; (2) placing sounds outsidethe stereo basis and (3) complete 3D simulation. Sound can becomplementary to haptics because little kinesthetic information extendsinto the audio frequency range. Audio may play an important role inperceiving the important information generated by the surgical site orthe OR by the surgeon located in a remote site.

FIG. 16 is a block diagram depicting certain aspects of an exemplarysoftware architecture that can be used with the local surgical cockpits2 and remote surgical procedures herein. In this example, the softwarearchitecture includes two layers: high level software 92 and low levelsoftware 94. The software comprises multiple interfaces to peripheralhardware and software components such as seat 8, display 50 and 3-Daudio input devices 80. The software can also be used for the surgicalcockpit 2 to communicate with any desired surgical module (surgicalrobot) that shares a proper communication channel such as a common anduniversally accepted Transmission Control Protocol/Internet Protocol(TCP/IP). Thus, the surgical cockpit can not only be used with nearbyremote surgical sites but also with distant remote surgery sites locatedoutside at least one, and possibly any, building containing at least oneof the surgical cockpit(s).

The high level software 92 can comprise a software layer that smoothlyinterfaces the teleoperation capability with image-guided softwaremodules residing either at the remote or local surgeon site, and asoftware module that facilitates collaborative communication amongmultiple surgical consoles. The high-level software module 92 can mergethe information provided by the hand, foot, and peripheral input device34. Image data and vital signs are presented continuously and/or throughcomputer generated audio cues. The high level module can be divided intoprocesses, and if desired each process can be dedicated to one of theperipheral elements depicted in FIG. 8.

The low level software 94 can comprise a robust software layer thatleverages control techniques to support haptic feedback via Internetconnections at local, national, and global scales. The low-levelsoftware module can be responsible for the interpretation of thelow-level input signals acquired from the hand and feet interfaces.Signals are translated to and from the surgical robot over a networkusing TCP/IP.

Network communications can update the software modules, for example thehigh level module that manages visual and audio display.

The low level software can be primarily a real-time segment of thesoftware running on a RT Linux operating system. The high level modulecan be non real-time software.

If desired, the interaction between the high level and low level modulescan be primarily unidirectional, in which position and orientation ofthe surgical tool location can be sent from the low level to the highlevel, with the only exception in which information can be sent back tothe low level module being when a tool reaches a position that exceeds apredefined safe zone.

As may be seen from the foregoing, the present invention provides animproved surgical device that permits surgical instruments to reachremote portions of the body with reduced trauma. The device sheath maybe steered to a surgical site around sensitive or critical tissue. Thesurgical tool components may be removed for replacement or cleaningwithout the device having to be straightened or removed from the body.Further, the tool deflection assemblies and methodology renders precisecontrol of the surgical tool components in all required degrees anddirections of movement. The present invention is thus well suited foruse in many different applications, including robotic surgical systems.

From the foregoing, it will be appreciated that, although specificembodiments have been discussed herein for purposes of illustration,various modifications may be made without deviating from the spirit andscope of the discussion herein. Accordingly, the systems and methods,etc., include such modifications as well as all permutations andcombinations of the subject matter set forth herein and are not limitedexcept as by the appended claims or other claim having adequate supportin the discussion and figures herein.

1. A local surgical cockpit comprising: a base, a frame disposed on thebase, a seat for an operator disposed on the frame, and a remotesurgical console configured such that the operator can operate theconsole for remote surgery while in the seat, wherein the seat isergonomic and is operably connected to the frame such that the seat canbe retainably tilted from a substantially upright position to asubstantially supine position.
 2. The local surgical cockpit of claim 1wherein the seat comprises an independently movable headrest, backrest,seating plate and footrest, the seat further comprising positioningelements operably connected to the independently movable headrest,backrest, seating plate and footrest and providing at least three axesof retainable positioning movement for each of the independently movableheadrest, backrest, seating plate and footrest.
 3. The local surgicalcockpit of claim 2 wherein the seat includes a lumbar support comprisingretainable positioning movement for support of the lower back.
 4. Thelocal surgical cockpit of claim 1 or 2 wherein a reference body postureof the seat corresponds to a human body posture that is fully relaxed inmicro gravity.
 5. The local surgical cockpit of claim 1 or 2 wherein thecockpit further comprises at least one peripheral device operablyconnected to move with the seat when the seat is moved so that thelocation of the peripheral device relative to the operator in the seatis substantially unchanged.
 6. The local surgical cockpit of claim 4wherein the peripheral device is at least one of a monitor facing anoperator in the seat and operably linked to display a remote surgicalsite, a heads-up display disposed in front of the local operator's eyes,and an input device disposed at a hand of the operator and operablylinked to provide input to a corresponding device located at the remotesurgical site.
 7. The local surgical cockpit of claim 6 wherein thecockpit comprises all of the monitor, the heads-up display and the inputdevice.
 8. A local surgical cockpit comprising: a local surgical cockpitcomprising a local surgical console configured for transmitting surgicalmovements of local operator operating the local surgical console to aremote surgery site, and a head-mounted display disposed in front of thelocal operator's eyes in surgical position in the cockpit to operate theconsole for surgery, wherein the head-mounted display is configured todepict at least images of a remote surgical site under remote operationby the operator.
 9. The local surgical cockpit of claim 8 wherein thelocal surgical cockpit is part of a system and the system furthercomprises remote image sensors operably connected to the head-mounteddisplay to transmit the image of the remote surgical site.
 10. The localsurgical cockpit of claim 8 wherein the head-mounted display extends tothe local operator's eyes from an articulated boom disposed in front ofthe local operator's eyes.
 11. The local surgical cockpit of claim 8wherein the articulated boom is actuated by at least one hand controllocated on the cockpit.
 12. The local surgical cockpit of claim 8wherein the boom is actuated by voice control.
 13. The local surgicalcockpit of claim 8 wherein the head-mounted display is disposed on ahead-mounted frame configured to rest on an operator's head and tomaintain the images in front of the local operator's eyes when theoperator's head moves.
 14. The local surgical cockpit of claim 8 whereinthe head-mounted display comprises two separate streams of videodisplayed to each eye of the local operator's eyes, each streamcomprising corresponding right and left eye views of a remote surgicalsite to provide a 3-D image of the site.
 15. The local surgical cockpitof claim 8 wherein the cockpit further comprises at least one monitor.16. A local surgical cockpit comprising: a local surgical cockpitcomprising a local surgical console configured for transmitting surgicalmovements of an operator operating the local surgical console to aremote surgery site, and comprising at least one image display deviceconfigured to depict at least one image of the remote surgical site, thedisplay device further depicting augmented reality for the operatorcomprising augmented information shown on the display and superimposedover the image of the remote surgical site.
 17. The local surgicalcockpit of claim 16 wherein the local surgical cockpit is part of asystem and the system further comprises remote image sensors operablyconnected to the head-mounted display to transmit the image of theremote surgical site.
 18. The local surgical cockpit of claim 16 whereinthe augmented information comprises at least one of preselected marginsto dissect during the surgery and a mask of vital structures in theremote surgical site overlaid over the images of the remote surgicalsite.
 19. The local surgical cockpit of claim 16 or 18 wherein thedisplay device further displays further augmented information either toa side of or superimposed over the image of the remote surgical site andthe further augmented information comprises at least one of bloodpressure, temperature, O₂ level, CO₂ level, intracranial pressure, apreplanned trajectory for a surgical tool, tool type, suction on/off, abottom task bar, recording capabilities, current time, and elapsed time.20. The local surgical cockpit of claim 16 or 18 wherein the image ofthe remote surgical site and the augmented information comprise blendinggraphical images with real-world views of the remote surgical site. 21.The local surgical cockpit of claim 16 or 18 wherein the image of theremote surgical site is provided by at least one of an endoscopiccamera, a remote surgical site camera, or a camera showing an operatingroom.
 22. A local surgical cockpit comprising: a local surgical cockpitcomprising a local surgical console configured for transmitting surgicalmovements of an operator operating the local surgical console to aremote surgery site, and comprising a local surgical instrumentcomprising local input surgical fingers configured to provide input tocorresponding remote surgical fingers configured to manipulate a remotesurgical instrument at a remote operation site, wherein the localfingers are high frequency haptic fingers configured to provide tactilefeedback to the operator based on acceleration of the remote surgicalinstrument manipulated by the remote surgical fingers.
 23. The localsurgical cockpit of claim 22 wherein the local surgical cockpit is partof a system and the system further comprises the remote surgicalfingers, and wherein the remote surgical fingers are haptic fingersconfigured to provide tactile feedback to the operator based onacceleration of the remote surgical instrument manipulated by the remotesurgical fingers.
 24. The local surgical cockpit of claim 23 wherein thelocal surgical cockpits of the system are configured such that operatorsin different locales can operate simultaneously on a single surgicalsite.
 25. The local surgical cockpit of claim 24 wherein the localsurgical cockpits of the system are configured such that operators canrelieve each other in a single surgery at a single surgical site. 26.The local surgical cockpit of claim 23 wherein the local surgicalcockpits of the system are configured to provide a teaching surgicalcockpit and a student surgical cockpit providing haptic feedback to astudent operator generated by a teaching operator.
 27. The localsurgical cockpit of claim 23 wherein the haptic feedback to the studentcomprises movements of a remote surgical instrument controlled by theteaching operator.
 28. The local surgical cockpit of claim 23 whereinthe haptic feedback to the student comprises tactile feedback from asurgical site being operated on by the teaching operator.
 29. A localsurgical cockpit comprising: a local surgical cockpit comprising a localsurgical console configured for transmitting surgical movements of anoperator operating the local surgical console to a remote surgery site,and comprising at least seven degrees of freedom for a local surgicalinstrument manipulated by a robotic arm manipulated by the operator,wherein the console is configured such that the seven degrees of freedomare transmissible to a remote surgical instrument located at a remotesurgical site and manipulated by the operator operating the console. 30.The local surgical cockpit of claim 29 wherein the local surgicalcockpit is part of a system and the system further comprises the remotesurgical instrument operably connected to the local surgical instrumentsuch that the remote surgical instrument precisely responds in at leastseven corresponding degrees of freedom to movements of the localsurgical instrument.
 31. The local surgical cockpit of claim 29 whereinthe degrees of freedom comprise at least nine degrees of freedom for thelocal surgical instrument manipulated by the operator and acorresponding nine degrees of freedom for the remote surgicalinstrument.
 32. The local surgical cockpit of claim 29 wherein thedegrees of freedom comprise at least twelve degrees of freedom for thelocal surgical instrument manipulated by the operator and acorresponding twelve degrees of freedom for the remote surgicalinstrument, wherein the local robotic arm comprises a shoulder joint, anelbow joint, a wrist joint and the three fingers, each comprising atleast the following degrees of freedom: shoulder comprises 2 degrees offreedom; elbow comprises 1 degree of freedom; wrist comprises 3 degreesof freedom; the three fingers comprise 2 degrees of freedom each. 33.The local surgical cockpit of claim 29 wherein local surgical instrumentcomprises at least three input fingers configured to provide input to acorresponding at least three remote surgical fingers configured tomanipulate a remote surgical instrument at a remote operation site,wherein the at least three input fingers are configured to bemanipulated by a single hand of an operator operating the local surgicalinstrument, and wherein the at least seven degrees of freedom compriseat least two degrees of freedom for two of the three remote surgicalfingers and at least three degrees of freedom for a third of the threeremote surgical fingers.
 34. The local surgical cockpit of claim 29wherein local surgical instrument comprises at least three input fingersconfigured to provide input to a corresponding at least three remotesurgical fingers configured to manipulate a remote surgical instrumentat a remote operation site, wherein the at least three input fingers areconfigured to be manipulated by a single hand of an operator operatingthe local surgical instrument, and wherein the degrees of freedomcomprise at least nine degrees of freedom comprising at least threedegrees of freedom for each of the three remote surgical fingers. 35.The local surgical cockpit of claim 29 wherein local surgical instrumentcomprises at least three input fingers configured to provide input to acorresponding at least three remote surgical fingers configured tomanipulate a remote surgical instrument at a remote operation site,wherein the at least three input fingers are configured to bemanipulated by a single hand of an operator operating the local surgicalinstrument, wherein the local robotic arm comprises a shoulder joint, anelbow joint, a wrist joint and the three fingers, each comprising atleast the following degrees of freedom: shoulder comprises 2 degrees offreedom; elbow comprises 1 degree of freedom; wrist comprises 3 degreesof freedom; the three fingers comprise 2 degrees of freedom each. 36.The local surgical cockpit of claim 35 wherein the three fingerscomprise 3 degrees of freedom each.
 37. The local surgical cockpit ofclaim 29 wherein the degrees of freedom provide for variable desiredpositioning and orientation of a tip of the remote surgical instrumentin space in 6 parameters including Cartesian position (x,y,z), andangular orientation (x y z θ, θ, θ).
 38. The local surgical cockpit ofclaim 29 wherein control of the remote surgical instrument furthercomprises scaling factors configured such that motion input by theoperator is attenuated or amplified with respect to the remote surgicalinstrument.
 39. The local surgical cockpit of claim 29 wherein controlof the remote surgical instrument further comprises indexing configuredto allow the operator to disengage the input device from the remotesurgical instrument to reposition his/her arms and engage again.
 40. Alocal surgical cockpit comprising: a local surgical cockpit comprising alocal surgical console configured for transmitting surgical movements ofan operator operating the local surgical console to a remote surgerysite, and comprising a local surgical instrument comprising at leastthree input fingers configured to provide input to a corresponding atleast three remote surgical fingers configured to manipulate a remotesurgical instrument at a remote operation site, wherein the at leastthree input fingers are configured to be manipulated by a single hand ofan operator operating the local surgical instrument.
 41. The localsurgical cockpit of claim 40 wherein the local surgical cockpit is partof a system and the system further comprises the three remote surgicalfingers operably connected to the three input fingers such that thethree remote surgical fingers precisely respond to movements of thethree input fingers.
 42. The local surgical cockpit of claim 40 whereinthe at least three input fingers are configured to correspondrespectively to a) an operator's thumb, b) an operator's index andmiddle fingers, and c) an operator's ring and little fingers.
 43. Thelocal surgical cockpit of claim 40 wherein the at least three inputfingers are configured to correspond respectively to a) an operator'sthumb, b) an operator's index finger, and c) an operator's middle, ringand little fingers.
 44. The local surgical cockpit of claim 40 whereinthe at least three input fingers are haptic fingers configured toprovide tactile feedback to the operator based on acceleration of aremote surgical instrument manipulated by the remote surgical fingers.45. The local surgical cockpit of claim 40 wherein the three inputfingers are operably connected so that two fingers control remotesurgical fingers and the remaining third finger controls an externaldevice.
 46. The local surgical cockpit of claim 40 wherein the externaldevice is a one or more of an electrocautery device, a laserphotocoagulator, a staple applier.
 47. The local surgical cockpit ofclaim 40 wherein the external device is an optical aspect of the camerasystem such as focus, zoom, rotation, or field-of-view.
 48. A localsurgical cockpit comprising: a local surgical cockpit comprising a localsurgical console configured for transmitting surgical movements of anoperator operating the local surgical console to a remote surgery site,and comprising at least one haptic foot pedal configured to be operablyconnected to at least one remote device at a remote operation site,wherein the at least one haptic foot pedal is configured to bemanipulated by a foot of the operator operating the local surgicalconsole to cause a movement or control change in the remote device. 49.The local surgical cockpit of claim 48 wherein the local surgicalcockpit is part of a system and the system further comprises the atleast one remote device operably connected to the at least one hapticfoot pedal.
 50. The local surgical cockpit of claim 48 wherein the localsurgical cockpit further comprises at least two haptic foot pedalsconfigured to be operably connected to the at least one remote device atthe remote operation site.
 51. The local surgical cockpit of claim 48wherein the local surgical cockpit further comprises at least eighthaptic foot pedals configured to be operably connected to at least tworemote devices at the remote operation site, the at least eight hapticfoot pedals divided to provide at least a first foot pedal set andsecond foot pedal set, wherein a first foot pedal set and second footpedal set are each disposed to be manipulated by a right foot of theoperator and by a left foot of the operator, respectively.
 52. The localsurgical cockpit of claim 51 wherein, for each of the first foot pedalset and second foot pedal set, the sets each contain four pedals witheach of the four pedals in one of four quadrants of a circle.
 53. Thelocal surgical cockpit of claim 51 wherein opposed pairs of the fourpedals are assigned opposed functions at the remote surgical site. 54.The local surgical cockpit of claim 51 wherein the opposed functions aresuction and irrigation.
 55. The local surgical cockpit of claim 51wherein each of the four pedals is assigned complementary functions fora remote instrument at the remote surgical site.
 56. The local surgicalcockpit of claim 55 wherein each of the four pedals controls the viewingangles of an endoscopic camera.
 57. The local surgical cockpit of claim48 wherein the local surgical cockpit further comprises a dead zone thatprevents two opposing functions being implemented simultaneously. 58.The local surgical cockpit of claim 48 wherein the at least one hapticfoot pedal controls at least one of camera angle, camera zoom, camerafocus, irrigation, suction, robot brakes, electric coagulation, laserphotocoagulation.
 59. A local surgical cockpit comprising: a localsurgical cockpit comprising a local surgical console configured fortransmitting surgical movements of an operator operating the localsurgical console to a remote surgery site, and comprising at least onevirtual console control knob presented virtually to the operator andconfigured to be manipulated by the operator to generate control signalsfor a corresponding remote device at a remote operation site.
 60. Thelocal surgical cockpit of claim 59 wherein the local surgical cockpit ispart of a system and the system further comprises the correspondingremote device.
 61. The local surgical cockpit of claim 60 wherein the atleast one virtual console control knob is a binary switch configured toprovide on/off signals to the corresponding remote device.
 62. The localsurgical cockpit of claim 60 wherein the at least one virtual consolecontrol knob is a gradual control knob configured to provide gradualcontrol signals to the corresponding remote device.
 63. The localsurgical cockpit of claim 59 wherein the virtual control knob isoperably connected to one of the three fingers of the haptic device. 64.The local surgical cockpit of claim 59 wherein the virtual control knobmust be virtually gripped by two or more fingers of the haptic devicebefore it may be rotated.
 65. A local surgical cockpit systemcomprising: at least two local surgical cockpits each comprising asurgical console configured for transmitting surgical movements of anoperator operating the local surgical console to a distant remotesurgery site located outside at least one building containing at leastone of the surgical cockpits, wherein each cockpit comprises arespective first and second set of at least two local robotic input armsconfigured to provide input to corresponding first and second remotesets of at least two corresponding remote robotic arms each configuredto manipulate a remote surgical instrument at a single remote operationsite, wherein the respective first and second set of local robotic inputarms are configured to be manipulated by respective first and secondoperators working in concert on the remote surgical site.
 66. The localsurgical cockpit system of claim 65 wherein the local surgical cockpitsystem is part of a further system and the further system furthercomprises the first and second remote sets of at least two correspondingremote robotic arms.
 67. The local surgical cockpit system of claim 65wherein the distant remote surgery site is located outside any buildingcontaining any of the local surgical cockpits.
 68. The local surgicalcockpit of claim 65 wherein the local surgical cockpits of the systemare configured such that operators in different locales can operatesimultaneously on a single surgical site.
 69. The local surgical cockpitof claim 65 wherein the local surgical cockpits of the system areconfigured such that operators can relieve each other in a singlesurgery at a single surgical site.
 70. The local surgical cockpit ofclaim 65 wherein the local surgical cockpits of the system areconfigured to provide a teaching surgical cockpit and a student surgicalcockpit providing haptic feedback to a student operator generated by ateaching operator.
 71. The local surgical cockpit of claim 70 whereinthe haptic feedback to the student comprises movements of a remotesurgical instrument controlled by the teaching operator.
 72. The localsurgical cockpit of claim 70 wherein the haptic feedback to the studentcomprises tactile feedback from a surgical site being operated on by theteaching operator.
 73. A local surgical cockpit comprising: a localsurgical cockpit comprising a local surgical console configured fortransmitting surgical movements of an operator operating the localsurgical console to a remote surgery site, and comprising at least fourlocal robotic input arms configured to provide input to a correspondingat least four remote robotic arms each configured to manipulate a remotesurgical instrument at a remote operation site, wherein the at leastfour local robotic input arms are configured to be manipulated by atleast one operator operating the local surgical console.
 74. The localsurgical cockpit of claim 73 wherein the local surgical cockpit is partof a system and the system further comprises the four remote roboticarms operably connected to the four local robotic input arms such thatthe four remote robotic arms precisely respond to movements of the fourlocal robotic input arms.
 75. The local surgical cockpit of claim 73wherein the four remote robotic arms are held in a sole arm-retentionstructure.
 76. The local surgical cockpit of claim 75 wherein the solearm-retention structure is configured to hold the four remote roboticarms such that the arms cannot collide with each other.
 77. The localsurgical cockpit of claim 73 wherein the cockpit is part of a systemcomprising at least two local surgical cockpits each configured for anoperator, and wherein the system is configured such that each operatorcan simultaneously hold a single remote robotic arm.
 78. The localsurgical cockpit of claim 73 wherein the cockpit is part of a systemcomprising at least two local surgical cockpits each configured for anoperator, and wherein the system is configured such that the operatorscan switch control of a remote robotic arm between each other.
 79. Alocal surgical cockpit comprising: a local surgical cockpit comprising alocal surgical console configured for transmitting surgical movements ofan operator operating the local surgical console to a remote surgerysite, and comprising local 3-dimensional audio configured to obtainremote 3-dimensional audio input from a remote operation site andprovide corresponding local 3-dimensional audio to an operator operatingthe console.
 80. The local surgical cockpit of claim 79 wherein thelocal surgical cockpit is part of a system and the system furthercomprises remote 3-dimensional audio sensors operably connected to thelocal 3-dimensional audio such that the local 3-dimensional audioprecisely transmit 3-dimensional audio signals from the remote3-dimensional audio sensors.
 81. The local surgical cockpit of claim 79wherein the 3-dimensional audio signals are correlated with tactilefeedback to provide correlated response to haptic input devices at thelocal surgical cockpit.